Question answering system and question answering processing method

ABSTRACT

A question answering system estimates an answer type of an answer from a input question statement, extracts keywords from the question statement and retrieves a document database, extracts language expressions (answer candidates) from the extracted document data, assigns evaluation points thereto. When there are a plurality of answer candidates having the same language expression, the system sorts those evaluation points in descending ranking order of evaluation, calculates values of evaluation points using such weighting that the value to be processed for each evaluation point diminishes, regards the sum total of those values as the evaluation point of the answer candidate, and outputs the answer candidate whose counted evaluation point is equal to or greater than a predetermined evaluation as an answer.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a natural language processing system using acomputer to operate as a question answering system for outputting ananswer to a question statement expressed in a natural language. Morespecifically, the present invention relates to a question answeringsystem which adds up evaluation points of a plurality of answercandidates having the same language expression when extractingcandidates of answers to the question, obtains the evaluation points andoutputs an answer candidate with a higher evaluation point than apredetermined point assigned as an answer.

The question answering system refers to a system which outputs, when aquestion in a natural language is input, an answer to the questionitself. For example, suppose a question “which part of the brain whosecells are dead is related to symptoms of Parkinson's disease?” is inputto the question answering system. The question answering system findsout a statement “Parkinson's disease is the to be caused when melanocytein the substantia nigra of the midbrain denatures and dopamine which isa neurotransmitter created in substantia nigra cells is lost” from amassive amount of digitized text including data such as Web pages, newsarticles, encyclopedia and outputs an answer “substantia nigra”precisely.

Since the question answering system can extract an answer not from alogical expression or database but from a plain statement (text data)written in a natural language, it is possible to use a massive amount ofexisting document data.

Furthermore, unlike an information retrieval system which the user needsto find out an answer from articles retrieved using keywords, thequestion answering system outputs a solution itself accurately, andtherefore the user can obtain information on the solution more quickly.

Furthermore, the question answering system automatically outputs thesolution itself, and therefore it can also be used as a knowledgeprocessing system inside another automatic knowledge processing systemand it is considered as a minimum necessary processing technology whenan artificial intelligence system is created.

Such a useful question answering system is considered to be a backbonesystem for intelligent processing and knowledge processing in the futureand great expectations are placed on the improvement of its processingcapacity.

2. Description of the Related Art

A general question answering system is roughly made up of threeprocessing means of answer expression estimation processing, documentretrieval processing and answer extraction processing.

The answer expression estimation processing is processing whichestimates an answer expression based on an expression of aninterrogative pronoun, etc., in a question statement entered. An answerexpression is a type of language expression of a desired answer. Thequestion answering system predetermines the correspondence of what kindof language expression of a question statement requires what kind ofanswer expression. Then, when the question statement entered is, forexample, “what is an approximate area of Japan?”, the question answeringsystem references the predetermined correspondence and estimates thatthe answer expression will be “numerical expression” from the expressionof “what is an approximate area” in the question statement. Furthermore,when the question statement is “who is Japan's prime minister?”, thequestion answering system estimates that the answer expression will be a“proper noun (personal name)” from the expression “who” in the questionstatement.

The document retrieval processing extracts keywords from the questionstatement, retrieves a document data group using the extracted keywordsand extracts document data in which the answer is considered to bewritten. When the question statement entered is, for example, “what isan approximate area of Japan?”, the question answering system extracts“Japan” and “area” as keywords from the question statement and retrievesdocument data including the extracted keywords “Japan” and “area” fromvarious document data groups to be retrieved.

The answer extraction processing extracts a language expression thatmatches the estimated answer expression from the document data includingkeywords extracted through the document retrieval processing and outputsthe language expression as an answer. The question answering systemextracts the language expression corresponding to the “numericalexpression” estimated through the answer expression estimationprocessing from the document data including the keywords “Japan” and“area” retrieved through the document retrieval processing as an answer.

Through the above described processing, in response to a questionstatement “what is the capital of Japan?”, the question answering systemoutputs an answer “Tokyo.” Nowadays, there is also a question answeringsystem in which when an answer is output, points (evaluation points) forevaluating answer candidates such as a degree of matching are assignedto answer candidates and an answer candidate which has acquiredpredetermined evaluation points is output as an answer. For example,suppose when evaluation points are assigned to answer candidates for thequestion statement “what is the capital of Japan?”, “rank; answercandidate; evaluation point; document data identification information(document number) from which the answer candidate is extracted” areoutput as answer candidate data as follows:

-   -   1; Kyoto; 3.3; document number 134,    -   2; Tokyo; 3.2; document number 12,    -   3; Tokyo; 2.8; document number 455,    -   4; Tokyo; 2.5; document number 371,    -   5; Tokyo; 2.4; document number 221,    -   6; Beijing; 2.2; document number 113

Then, when the question answering system adopts the first rank answercandidate and outputs “Kyoto” as an answer, a wrong answer is outputbecause the correct answer is “Tokyo.”

Thus, within the document data which becomes the answer retrievaltarget, language expressions appearing at many locations together withthe expression relating to the content of the question are considered tohave more relatedness with regard to the question and can be consideredto match the answer of the question better. Based on this concept, foranswer candidates having the same language expressions appearing indifferent document data or at different locations in the document data,there is a technique of adding up evaluation points of the respectiveanswer candidates and regarding the sum total as the evaluation point ofthe answer candidate (for example, see Reference 1).

-   [Reference 1: Toru Takaki, Yoshio Eriguchi, “NTTDATA    Question-Answering Experiment at the NTCIR-3 QAC”, National    Institute of Informatics, The NTCIR Workshop 3 Meeting (3rd NTCIR    workshop meeting), October 2002, p. 95-100]

For example, in the example of the aforementioned answer candidate forthe question statement “what is the capital of Japan?”, evaluationpoints of answer candidates are simply added up and counted using theconventional technique. When evaluation points given to the answercandidates “Tokyo” appearing in four document data pieces or at fourlocations out of the aforementioned answer candidates are counted andregarded as the evaluation point of the answer candidate “Tokyo”, theevaluation ranking of each answer candidate for the question statementis as follows:

-   -   1; Tokyo; 10.9; document number 12,455,371,221,    -   2; Kyoto; 3.3; document number 134,    -   3; Beijing; 2.2; document number 113

Then, since the first rank answer candidate “Tokyo” is adopted in thequestion answering system, the answer output from the question answeringsystem is correct.

However, as shown in Reference 1 above, according to the conventionalart of simply adding up the evaluation points of the answer candidatesextracted from the document data which is the answer retrieval targetfor each answer candidate having the same language expression andadopting the answer candidate with an evaluation point equal to orhigher than a predetermined level assigned as the answer, there is aproblem that a language expression appearing with a high frequency inthe document data is likely to be selected as an answer and the accuracyof the answer does not necessarily improve.

Especially when a technique of simple evaluation point additionprocessing is applied to a question answering system with high accuracyof the answer candidate extraction processing itself, this problemappears more serious. In a question answering system which carries outhigh accuracy answer candidate extraction processing, though thereliability of evaluation points assigned through the originalprocessing is high, answer candidates are extracted by applying theconventional technique of adding evaluation points to this answercandidate extraction processing based on the total point simplycalculated from evaluation points of answer candidates. As a result,many answer candidates whose evaluation itself is low are evaluatedhigher, which leads to reduce the answering accuracy contrarily.

SUMMARY OF THE INVENTION

The present invention has been implemented in view of the abovedescribed problems and it is an object of the present invention toprovide a technique of extracting answer candidates capable of improvingthe accuracy of extracting answer candidates by counting evaluationpoints of the same answer candidate and reducing an adverse effect thatanswer candidates which appear with a high frequency are likely to beevaluated higher.

The present invention is a question answering system which receivesquestion statement data expressed in a natural language and outputs ananswer to the question statement data from a document data group whichis an answer retrieval target, comprising 1) answer type estimatingmeans for analyzing the language expression of the input questionstatement data and estimating an answer type which is a type of languageexpression which can be an answer to the question statement data, 2)document retrieving means for extracting keywords from the questionstatement data, retrieving and extracting document data including thekeywords from the document data group, 3) answer candidate evaluationpoint calculating means for extracting a language expression which canbe the answer from the document data as answer candidates and assigningevaluation points to the answer candidates, 4) answer candidateevaluation point counting means for sorting, when evaluation points arecounted for each answer candidate having the same language expression,the evaluation points of answer candidates having the same languageexpression in descending ranking order of evaluation, calculating valuesof the evaluation points using such weighting that the value processedfor each evaluation point diminishes as the rank of the evaluation pointdecreases and regarding the sum total of the values as the evaluationpoint of the answer candidate having the same language expression and 5)answer outputting means for outputting an answer candidate whoseevaluation point counted by the answer candidate evaluation pointcounting means is equal to or higher than a predetermined evaluationvalue as an answer.

The question answering system of the present invention analyzes alanguage expression of input question statement data and estimates theanswer type which is a type of language expression which can be ananswer to the question statement data. Then, it extracts keywords fromthe question statement data and retrieves and extracts document dataincluding the keywords from the document data group. Furthermore, itextracts a language expression which can be the answer from the documentdata as an answer candidate and assigns an evaluation point to theanswer candidate. Then, when evaluation points are counted for eachanswer candidate having the same language expression, the evaluationpoints of answer candidates having the same language expression aresorted in descending ranking order of evaluation ranking and the valuesof the evaluation points are calculated using such weighting that thevalue to be processed for each evaluation point diminishes as the rankof the evaluation point decreases. Then, the sum total of the values isregarded as the evaluation point of the answer candidate having the samelanguage expression and an answer candidate whose evaluation pointcounted by the answer candidate evaluation point counting means is equalto or higher than a predetermined evaluation value is output as ananswer.

Furthermore, the present invention is the question answering systemhaving the above described structure, wherein when counting evaluationpoints for each answer candidate having the same language expression,the answer candidate evaluation point counting means sorts and ranksevaluation points of answer candidates having the same languageexpression in descending ranking order of evaluation. Then, assumingthat n is the number of answer candidates having the same languageexpression and Point_(i) (1≦i≦n) is a ranking order of evaluation pointsof the answer candidates, it is possible to calculate Score using aformula 1;Score=Σ1/(i+b)/(i+b−1)*Point_(i)   (1)

-   -   where Σ is the sum total when i takes a value from 1 to n; from        which each sum total of the value of the answer candidates as        the representative evaluation point of the answer candidate        having the same language expression.

Or when counting evaluation points for each answer candidate having thesame language expression, the answer candidate evaluation point countingmeans sorts the evaluation points of answer candidates having the samelanguage expression in descending ranking order of evaluation, andassuming that n is the number of answer candidates having the samelanguage expression and Point_(i) (1≦i≦n) is a ranking order ofevaluation points of the answer candidates, it is possible to calculateScore using a formula 2;Score=Σk ^(i−1)*Point_(i)   (2)

-   -   where Σ is the sum total when i takes a value from 1 to n; from        which each sum total of the value of the answer candidates as        the representative evaluation point of the answer candidate        which is the same language expression.

According to the question answering system of the present invention,when evaluation points assigned to answer candidates having the sameextracted language expression are counted, the assigned evaluationpoints are sorted in descending order and values of the evaluationpoints are calculated using such weighting that the value processed oneach evaluation point diminishes as the rank of the evaluation pointdecreases and the evaluation points are added up. More specifically,when answer candidates are extracted, evaluation points of answercandidates having the same language expression are sorted in descendingorder of evaluation ranking, the value of each evaluation point P iscalculated using such weighting that the value diminishes as the rank ofthe evaluation point decreases and the sum total of the values ofweighted evaluation points P is calculated.

In this way, in the weighting processing of each answer candidate,evaluation point P is subtracted at a higher rate than that ofevaluation point P which is directly superior thereto. Therefore, thelower the initial evaluation of an answer candidate, the smaller theinfluence of the counted evaluation point on the sum total becomes, andtherefore the influence of the answer extraction processing on theaccuracy of processing also decreases.

Furthermore, in the evaluation of an answer candidate, it is possible toreduce an adverse influence that an answer candidate which appears witha high frequency is relatively likely to be evaluated higher, andtherefore the present invention exerts the effect of improving theprocessing accuracy of extracting answer candidates.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration example of a question answeringsystem according to the present invention;

FIG. 2 illustrates a processing flow of the present invention;

FIG. 3 illustrates processing accuracy in each processing example for apredetermined question setting;

FIG. 4 illustrates processing accuracy in each processing example for apredetermined question setting;

FIG. 5 illustrates processing accuracy in each processing example for apredetermined question setting;

FIG. 6 illustrates processing accuracy in each processing example for apredetermined question setting; and

FIG. 7 illustrates processing accuracy in each processing example for apredetermined question setting.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 shows a configuration example of a question answering system 1 ofthe present invention. The question answering system 1 is provided witha question statement input section 10, an answer type estimation section20, a document retrieval section 30, an answer candidate extractionsection 40, an answer output section 50 and a document database 2.

The question statement input section 10 is the means for inputting aquestion statement expressed in a natural language.

The answer type estimation section 20 is the means for analyzing thequestion statement input by the question statement input section 10 andestimating the answer type of an answer to be output based on an answertype estimation rule. Suppose the answer type and answer type estimationrule are prepared beforehand. The answer type is the type of a languageexpression which can be an answer to be output and, for example, typesof proper nouns such as a personal name, geographic name, organizationname, numerical expression such as quantity and sum of money and timeexpression are set. The answer type estimation rule is a heuristic rulewhich estimates the corresponding answer type of the answer according tothe expression such as expression of a question statement or expressionof an interrogative included in the question statement and, for example,the following rules are defined:

-   -   1) When the question statement includes an expression “who,” the        answer type is “personal name.”    -   2) When the question statement includes an expression “when,”        the answer type is “time expression.”    -   3) When the question statement includes an expression “how many        (or much),” the answer type is “numerical expression.”

The document retrieval section 30 is the means for retrieving andextracting document data including keywords from the document database 2which is the answer retrieval target using keywords extracted from thequestion statement entered by the question statement input section 10.The document retrieval section 30 uses a generally known documentretrieval technique.

The answer candidate extraction section 40 is the means for extracting alanguage expression to be a possible answer with an evaluation pointassigned from the document data retrieved by the document retrievalsection 30 as an answer candidate, deciding the answer type of theextracted answer candidate, and further counting the evaluation point ofthe answer candidate. The answer candidate extraction section 40 isconstructed of an answer candidate selection section 41, an answercandidate/keyword proximity evaluation section 42, an answer candidateanswer type decision section 43, an answer candidate evaluation pointcalculation section 44 and an answer candidate evaluation point countingsection 45.

The answer candidate selection section 41 is the means for extracting alanguage expression to be a possible answer from the document dataretrieved by the document retrieval section 30 and generating an answercandidate. The answer candidate/keyword proximity evaluation section 42stochastically evaluates proximity between an answer candidate withinthe document data of an extraction source and keywords and assigns anevaluation point p₁ based on the proximity to the answer candidate.

The answer candidate answer type decision section 43 is the means fordeciding the answer type of the answer candidate generated by the answercandidate selection section 41 based on a predetermined answer typedecision rule and assigning an evaluation point p₂ according to thedecided answer type to the answer candidate. The answer type decisionrule is the rule that defines the correspondence between answer types asthe answer type estimation rule. When the decided answer type is thesame as the answer type estimated by the answer type estimation section20, the answer candidate answer type decision section 43 evaluates itparticularly high and evaluates higher as the relatedness to theestimated answer type becomes stronger.

The answer candidate evaluation point calculation section 44 is themeans for totaling evaluation points p₁ assigned by the answercandidate/keyword proximity evaluation section 42 and evaluation pointsp₂ assigned by the answer candidate answer type decision section 43 foreach answer candidate to calculate an evaluation point P.

When there are a plurality of answer candidates having the same languageexpression in the extracted answer candidates, the answer candidateevaluation point counting section 45 is the means for countingevaluation points for each answer candidate having the same languageexpression. More specifically, the answer candidate evaluation pointcounting section 45 sorts evaluation points P of answer candidateshaving the same language expression in descending ranking order ofevaluation, assigns weights to the respective evaluation points P insuch a way that evaluation points P to be added diminish and calculatesthe sum total of the weighted evaluation points P. In the weightingprocessing of each answer candidate, an evaluation points P is added asa value subtracted at a greater rate than that of the evaluation point Pimmediately superior thereto. Thus, the lower the initial evaluationpoint of an answer candidate, the smaller the influence on the sum totalof the counted evaluation points becomes, and therefore the influence ofthe original answer extraction processing on the processing accuracyalso decreases.

The answer output section 50 is the means for outputting an answercandidate with a higher evaluation point than a predetermined level asan answer from among answer candidates extracted by the answer candidateextraction section 40 and with evaluation points assigned.

FIG. 2 shows a processing flow of the present invention. The questionstatement input section 10 of the question answering system 1 receives aquestion statement (step S10). The answer type estimation section 20estimates the answer type from the expression of the question statement(step S11). The answer type estimation section 20 carries out amorphological analysis on the question statement entered, references apredetermined answer type estimation rule based on the expression of ananalyzed interrogative pronoun, etc., and estimates the answer type ofthe answer to the question statement. For example, when the questionstatement entered is “what is an approximate area of Japan?”, the answertype estimation section 20 extracts the expression “what is anapproximate area” from the question statement, references theaforementioned answer type estimation rule and estimates that the answertype is “numerical expression.”

Then, the document retrieval section 30 extracts keywords from thequestion statement (step S12), retrieves the document database 2 usingthe extracted keywords and extracts document data including the keywords(step S13). When the question statement entered is “what is anapproximate area of Japan?”, the document retrieval section 30 carriesout a morphological analysis on the question statement, extracts nouns“Japan, area” from the question statement as keywords. It retrieves thedocument database 2 using the keywords “Japan, area” and extracts thedocument data including the keywords “Japan, area.” As a result ofretrieval, the extracted document data becomes the target from which theanswer to the question statement is extracted.

Then, the answer candidate selection section 41 of the answer candidateextraction section 40 extracts an answer candidate having the languageexpression which can be an answer from the extracted document data (stepS14). For example, the answer candidate selection section 41 extracts acharacteristic string “n-gram” from the extracted document data andextracts language expressions decided to be noun phrases, unknownphrases, symbols, etc., from among the language expressions as answercandidates.

Furthermore, the answer candidate/keyword proximity evaluation section42 decides the proximity in the location of appearance between theextracted answer candidate and keyword in the document data and assignsevaluation point p₁ to the answer candidate (step S15). Here, it isassumed that within the retrieved document data, as an answer candidateappears closer to the keyword, that is, the answer candidate and keywordappear within a narrower range, the answer candidate and keyword have ahigher level of mutual relatedness and an answer candidate having a highlevel of relatedness to the keyword is better as the answer to thequestion statement. Then, the answer candidate/keyword proximityevaluation section 42 assigns a higher evaluation point p₁ as the answercandidate appears closer to the keyword.

Furthermore, the answer candidate answer type decision section 43decides the answer type of an answer candidate extracted from theprocessing in step S14 and assigns evaluation point p₂ (step S16). Theanswer candidate answer type decision section 43 references apredetermined answer type decision rule, decides the answer type of theanswer candidate and assigns evaluation point p₂ to the answer candidateaccording to the decided answer type. When it is decided that the answertype of the answer candidate is the same as the answer type estimated inthe processing of step S11, the answer candidate answer type decisionsection 43 assigns the highest evaluation point (e.g., 1,000 points) tothe answer candidate and when the language expression of the answercandidate is a language expression which cannot be the answer type, itassigns a minus evaluation point (e.g., −1,000,000) to the answercandidate.

Furthermore, the answer candidate evaluation point calculation section44 calculates an evaluation point P from the evaluation point p₁ andevaluation point p₂ for each answer candidate (step S17). The evaluationpoint calculation section 44 totalizes the evaluation point p₁ assignedthrough the processing in step S15 and the evaluation point p₂ assignedthrough the processing in step S16 for each answer candidate as theevaluation point P of the answer candidate.

Then, the answer candidate evaluation point counting section 45totalizes evaluation points P for each answer candidate having the samelanguage expression (step S18). When there are a plurality of answercandidates having the same language expression in the answer candidatesextracted through the processing in step S14, the answer candidateevaluation point counting section 45 gathers those answer candidatesinto one and assigns an evaluation point Score as one answer candidate.

The answer candidate evaluation point counting section 45 sorts therespective evaluation points P of the answer candidates having the samelanguage expression in descending ranking order of evaluation as below:Point_(i)(1≦i≦n),Point_(i)(1≦i≦n),

-   -   where n=number of answer candidates having the same language        expression.

Then, the evaluation points P of answer candidates are added up usingeither one of the following two methods to count the evaluation pointScore.

(1) First Counting MethodScore=Σ(1+b)*b/(i+b)/(i+b−1)*Point_(i)  (Formula 1)

-   -   (Σ is the sum total when i takes a value from 1 to n).    -   Here, “(1+b)*b” is a constant and has no influence on the order        relationship, and therefore it is omissible.        (2) Second Counting Method        Score=Σk ^(i−1)*Point_(i)  (Formula 2)    -   (Σ is the sum total when i takes a value from 1 to n)    -   Here, the evaluation point Score is the sum total of weighted        evaluation points P and weights of the respective answer        candidates are subtracted at a greater rate than weights for the        evaluation point P of immediately superior thereto and added.

Then, the answer output section 50 selects an answer candidate assignedan evaluation point Score equal to or greater than a predetermined valueand outputs it as an answer (step S19).

Furthermore, the present invention will be explained using anotherembodiment.

In another embodiment, a question answering system 1′ of the presentinvention has substantially the same structure of the processing meansas that shown in FIG. 1, but it is provided with an answer typeestimation section 20′ and an answer candidate answer type decisionsection 43′ instead of the answer type estimation section 20 and answercandidate answer type decision section 43.

Instead of carrying out processing using heuristic rules to estimate ordecide a predetermined answer type, the answer type estimation section20′ and answer candidate answer type decision section 43′ of thequestion answering system 1′ estimate or decide an answer type using asupervised machine learning method. In this case, predetermined answertypes are prepared and patterns of pairs of a correct input (questionstatement) and output (estimated answer type) are manually preparedbeforehand as teaching data (learning data) for each problem.

The answer type estimation section 20′ receives teaching data which is apair of a question statement and estimated answer type first and thenanalyzes the question statement into predetermined features, learns whatanswer type it is likely to become in what kind of question statementfeatures through machine learning processing and stores the learningresult. Then, the answer type estimation section 20′ likewise analyzesthe question statement (input) entered from the question statement inputsection 10 into features, obtains the probability that it may become theanswer type from the learning result for each feature and estimates theanswer type with the highest probability as the answer type (output) ofthe question statement.

As with the answer type estimation section 20′, the answer candidateanswer type decision section 43′ receives an answer candidate andteaching data paired with the answer type decided, learns what answercandidate is likely to become what answer type through machine learningprocessing and stores the learning result. Then, the answer candidateanswer type decision section 43′ determines the probability that eachanswer candidate generated by the answer candidate selection section 41is likely to become each answer type and estimates the answer type withthe highest probability as the answer type of the answer candidate.

The machine learning method used by the answer type estimation section20′ or answer candidate answer type decision section 43′ includes, forexample, a maximum entropy method and support vector machine method orthe like.

The maximum entropy method is a processing technique that determines aprobability distribution whose entropy becomes a maximum on conditionthat an expected value of appearance of a feature which is a small unitof information used to estimate learning data is equal to an expectedvalue of appearance of a feature of unknown data, determines based onthe probability distribution determined a probability that eachappearance pattern of the feature may fall under each category andregards the category having the maximum probability as the category tobe determined.

The support vector machine method is a technique that classifies dataconsisting of two categories by splitting a space by a hyperplane andassuming a concept that the possibility of wrong classification inunknown data is lower for learning data having a greater distance(margin) between an instance group of two categories and the hyperplane,determines the hyperplane that maximizes this margin and carries outclassification using the hyperplane. When data having three or morecategories is classified, the data is processed by combining a pluralityof support vector machines.

In the case of a structure with the answer type estimation section 20′or answer candidate answer type decision section 43′ including a machinelearning processing function, only preparing learning data can realize ahighly versatile question answering system 1′ applicable to problems ofall descriptions.

Furthermore, the answer type estimation sections 20, 20′ and answercandidate answer type decision sections 43, 43′ may also be adapted soas to carry out either processing using a machine learning method with ateacher or processing using heuristic rules.

Furthermore, the answer type estimation sections 20, 20′ and answercandidate answer type decision sections 43, 43′ may also be adapted soas to carry out processing combining processing using a machine learningmethod with a teacher and processing using heuristic rules. That is, theanswer type estimation rule or answer type decision rule is adapted soas to include a rule that “a high evaluation point will be assigned whenthe estimation result of processing according to a machine learningmethod coincides with the estimation result/decision result obtainedbased on the answer type estimation rule or answer type decision rule.”

Note that the following Reference 2 describes a question answeringsystem using a machine learning method:

-   [Reference 2: Eisaku Maeda, “Question Answering Viewed from Pattern    Recognition/Statistical Learning”, Natural Language Understanding    and Methods of Communication, seminar material, Institute of    Electronics, Information and Communication Engineers, Natural    Language Understanding and Methods of Communication (NLC), Jan. 27,    2003, p. 29-64]

SPECIFIC EXAMPLE

A specific example of processing according to the present inventiontogether with a processing example using a conventional processingmethod will be explained below.

Suppose a first example and a second example will be processingaccording to a conventional method for a comparison with the presentinvention. In the first example, even if there are a plurality of answercandidates having the same language expression extracted from differentarticles, answers are evaluated without adding up evaluation points ofanswer candidates. In the second example, evaluation points of answercandidates having the same language expression are simply added up andthe answer candidates thereof are evaluated.

Suppose a third example and a fourth example are examples according tothe present invention. In the third example, evaluation points of answercandidates having the same language expression are counted according toa first counting method and the answer candidates are evaluated. In thefourth example, evaluation points of answer candidates are countedaccording to a second counting method and the answer candidates areevaluated.

As shown in the following Reference 3, in the first to fourth examples,assuming that the question statement to be entered is “what is anapproximate area of Japan?”, a news article corpus corresponding to newsarticles in 1998-1999 of Mainichi Newspaper Co., Ltd. was used as thedocument database 2 to be the answer retrieval target.

-   [Reference 3: Masaki Murata, Masao Utiyama, and Hitoshi Isahara, “A    Question-Answering System Using Unit Estimation and Probabilistic    Near-Terms IR”, National Institute of Informatics, NTCIR Workshop 3    Meeting (3rd NTCIR workshop meeting), 2002]

Furthermore, in each example of processing, a plurality of systems inwhich the respective processing means of the question answering system 1having the structure shown in FIG. 1 were provided to carry out thefollowing specific processing.

The first question answering system is the system which carries out theprocessing disclosed in Reference 2 and an outline of the processing isas follows (see Reference 3, Section 3.4).

Answer type estimation processing: a predetermined question statement isinput, a manually prepared answer type estimation rule is referenced andan answer type is estimated from expressions of a question statement.

Document retrieval processing: Furthermore, keywords are extracted froma question statement through morphological analysis processing and thedocument data of the above described news article corpus is retrieved asa complete statement without dividing it. The document data includingthe keywords is extracted.

Answer candidate extraction processing: Language expressions that can beanswers are extracted from the extracted document data, used as answercandidates and evaluation point p₁ is assigned to the answer candidatesusing a predetermined expression (here, Score_(near1) described inReference 2) based on the proximity of appearance between the answercandidates and keywords. Evaluation point p₁ is calculated in such a waythat higher evaluations are obtained as the answer candidates appearcloser to the keywords.

Furthermore, with reference to predetermined answer type decision rules,it is decided whether the answer candidates are similar to the answertype estimated by the answer type estimation processing or not andevaluation point p2 is assigned in such a way that higher evaluationsare obtained as the answer candidates are more similar to the estimatedanswer type. Especially when the answer candidates are of the sameanswer type as that estimated, particularly higher evaluation points(1,000 points) are assigned.

The answer type decision rule is adapted so as to include a rule thathigher evaluation points (1,000 points) are assigned when the processingresult according to the machine learning method provided withpredetermined teaching data coincides with the decision result accordingto the answer type decision rule and higher evaluation points (1,000points) are assigned to the answer candidates that satisfy this rule. Inthis way, the answer type of the answer candidate is decided through theprocessing by the machine learning method combined with the processingusing the heuristic rule.

The second question answering system executes the processingsubstantially the same as the first question answering system.

In the answer candidate extraction processing, the second questionanswering system does not carry out processing using meaningfulinformation on the answer candidates. That is, the answer type of theanswer candidate is not decided and no evaluation point p₂ of the answercandidate is calculated, and therefore an answer is extracted from amongthe analysis candidates based on only the evaluation point p₁.

In the answer candidate extraction processing, the third questionanswering system assigns evaluation points to the answer candidateswithout using proximity between the answer candidates and keywords basedon the appearance of the answer candidate and keyword within the samedocument data. Here, evaluation point p₁ will be calculated using anexpression with k=1 in Score_(near2) described in Reference 2.

The fourth question answering system carries out substantially the sameprocessing as the above described third question answering system. Inthe document retrieval processing, the fourth question answering systemdivides the document data which is the retrieval target into paragraphsand then carries out retrieval and decides the appearance of answercandidates and keywords paragraph by paragraph and carries outevaluations.

The fifth question answering system carries out substantially the sameprocessing as the above described first question answering system. Inthe document retrieval processing, the fifth question answering systemdivides the document data which is the retrieval target into paragraphsbeforehand and then retrieves them, decides proximity of appearancebetween the answer candidates and keywords paragraph by paragraph andcarries out evaluations.

The first example to the fourth example will be explained below.Processing was carried out using two problem setting tasks; Task 1 andTask 2. The total number of question statements was 200 each (seeReference 2).

FIRST EXAMPLE

In the first example, even if there were a plurality of answercandidates having the same language expression extracted from differentarticles in the first to fifth question answering systems, no evaluationpoints of answer candidates were added up and an answer was selectedbased on evaluation points assigned to the respective answer candidates.

SECOND EXAMPLE

In the second example, evaluation point counting processing was carriedout after answer candidate extraction processing.

In the evaluation point counting processing, the evaluation points P ofthe answer candidates having the same language expression extracted fromdifferent articles were simply added up as the evaluation points of theanswer candidates and an answer was selected based on the evaluationpoints assigned to the answer candidates.

Note that as described above, higher evaluation points (1,000 points)were assigned to the answer candidates which can be considered matchingthe estimated answer type, and therefore evaluations of the answercandidates drastically change in units of 1,000 points. For this reason,in order to prevent any influence on values of higher digits than thedigit of 1,000 (thousand) of evaluation points, values of lower digits(hundred) than the digit of 1,000 (thousand) are extracted from theevaluation points and the extracted values (evaluation points) are addedup. Then, the remaining values of digits equal to or higher than thedigit of 1,000 (thousand) of the evaluation points P of the answercandidates and the values of the digit (hundred) lower than the digit of1,000 (thousand) are added up as the evaluation points P of the answercandidates. Furthermore, the evaluation points of answer candidateshaving different values of the digit equal to or higher than the digitof 1,000 (thousand) are not added up.

For example, suppose two answer candidates A having the same languageexpression appear at two locations and the respective evaluation pointsare 1,025 and 1,016. Then, the evaluation point of the answer candidateA is 1,041. Furthermore, suppose the answer candidate B having the samelanguage expression appears at two locations resulting in evaluationpoints 2,025 and 2,016. Then, the evaluation point of the answercandidate B is 2,041. Furthermore, if the answer candidate C appears attwo locations and their respective evaluation points are 2,025 and1,016, the evaluation point of the answer candidate C is 2,025.

THIRD EXAMPLE

In the third example, evaluation point counting processing is carriedout after answer candidate extraction processing. In the evaluationpoint counting processing, evaluation points P of answer candidateshaving the same language expression extracted from different articleswere counted based on the first counting method of the present inventionas the evaluation points of the answer candidates and an answer wasselected based on the evaluation point assigned to the answer candidate.

As in the case of the second example, the evaluation point P of theextracted answer candidate was divided at the digit of 1,000 (thousand)and values of lower digits were added up.

In the evaluation point counting processing, various values wereassigned to b of the following expression and evaluation point Score wascalculated:Score=Σ(1+b)*b/(i+b)/(i+b−1)*Point_(i)

-   -   (Σ is the sum total when i takes a value of 1 to n)

For example, when the answer candidate D appears at two locations andtheir respective evaluation points are 2,025 and 2,016.25 and 16 whichare the values of the respective evaluation points at the digit equal toor lower than the digit of 1000 are extracted as Point₁ and Point₂.

When b=1, the evaluation point of the final answer candidate D becomes2,030.33.

$\begin{matrix}{{Score} = {\sum{( {1 + b} )*{{b/( {i + b} )}/( {i + b - 1} )}*{Point}_{i}}}} \\{{Score} = {{( {1 + 1} )*{{1/( {1 + 1} )}/( {1 + 1 - 1} )}*25} +}} \\{{( {1 + 1} )*{{1/( {2 + 1} )}/( {2 + 1 - 1} )}*16}} \\{\mspace{59mu}{= 30.33}}\end{matrix}$

Furthermore, when b=0.1, the evaluation point of the final answercandidate D becomes 2025.76.

$\begin{matrix}{{Score} = {\sum{( {1 + b} )*{{b/( {i + b} )}/( {i + b - 1} )}*{Point}_{i}}}} \\{{Score} = {{( {1 + 0.1} )*{{0.1/( {1 + 0.1} )}/( {1 + 0.1 - 1} )}*25} +}} \\{{( {1 + 0.1} )*{{0.1/( {2 + 0.1} )}/( {2 + 0.1 - 1} )}*16}} \\{\mspace{59mu}{= 25.76}}\end{matrix}$

FOURTH EXAMPLE

In the fourth example, evaluation point counting processing was carriedout after answer candidate extraction processing. In the evaluationpoint counting processing, evaluation points P of answer candidateshaving the same language expression extracted from different articleswere counted based on the second counting method of the presentinvention as the evaluation point of the answer candidate and an answerwas selected based on the evaluation point assigned to the answercandidate.

As in the case of the second example, the evaluation point P of theextracted answer candidate was divided at the digit of 1,000 (thousand)and values at lower digits were added up.

In the evaluation point counting processing, various values wereassigned to k of the following expression and an evaluation point Scorewas obtained.Score=Σk ^(i−1)*Point_(i)

-   -   (Σ is the sum total when i takes a value of 1 to n)

For example, suppose the answer candidate D appears at two locations andthe respective evaluation points are evaluation points 2,025 and2,016.25 and 16 which are values at the digit equal to or lower than thedigit of 1,000 (thousand) of the respective evaluation points areextracted as Point₁ and Point₂. When k=0.1, the evaluation point of thefinal answer candidate D becomes 2,026.6.

$\begin{matrix}{{Score} = {25 + {16*0.1}}} \\{= 26.6}\end{matrix}$

Furthermore, when k=0.01, the evaluation point of the final answercandidate D becomes 2025.16.

$\begin{matrix}{{Score} = {25 + {16*0.01}}} \\{= 25.16}\end{matrix}$

As the evaluation technique in the first to fourth examples of the firstto fifth question answering systems, the processing accuracy of problemsetting tasks; Task1 and Task2 was evaluated using an evaluation scalecalled “MRR” for Task1 and an evaluation scale called “MF” for Task2.

The MRR causes the question answering system to output five solutions insequence and when a correct answer is obtained at the rth solution, theaccuracy of 1/r is obtained. Such a value was obtained in each questionstatement and this value divided by 200 question statements was regardedas the processing accuracy.

The MF is obtained by calculating the accuracy of each question withF-measure and averaging this with 200 question statements. F-measure isa reciprocal of the average of the reciprocal of a reproduction rate andthe reciprocal of an adaptation rate. The reproduction rate is thenumber of correct solutions of each question answering system divided bythe number of correct solutions. The adaptation rate is the number ofcorrect solutions of each question answering system divided by thenumber of answers output from the system.

In Task2, a problem is set in such a way as to allow questionstatements, for example “which countries are permanent members of theSecurity Council of the United Nations?” for which a plurality oflanguage expressions can be answers to be evaluated. Since Task2 allowsa plurality of answers to be output, the processing accuracy is examinedusing the evaluation scale of the MF. In the question answering systemhaving the structure such as the first to fifth question answeringsystems, it is possible to give a plurality of answers by consideringseveral higher level answers with high evaluation points or from theevaluation point of the highest answer to an answer with a predeterminedevaluation point difference as answers. However, for simplicity ofexplanations here, the first to fifth question answering systems have astructure of outputting only one highest answer.

FIG. 3 to FIG. 7 show processing accuracy in respective examplescorresponding to problem setting tasks; Task1 and Task2. FIG. 3illustrates processing accuracy in the first question answering system,FIG. 4 illustrates processing accuracy in the second question answeringsystem, FIG. 5 illustrates processing accuracy in the third questionanswering system, FIG. 6 illustrates processing accuracy in the fourthquestion answering system and FIG. 7 illustrates processing accuracy inthe fifth question answering system.

All the second to fifth question answering systems have a structure ofprocessing means having lower processing performance than the firstquestion answering system. This is done to confirm the effects of thepresent invention in the third example and fourth example also in thequestion answering system having inferior performance.

In the first question answering system, the processing accuracy in thesecond example in which evaluation points are simply added up was lowerthan that in the first example. However, in the third and fourthquestion answering systems, it is possible to confirm that the secondexample acquires higher processing accuracy than the first example.

In the third and fourth question answering systems with lowerperformance with no information on proximity between keywords andsolution candidates, simply adding up evaluation points increases theaccuracy. It is evident that depending on the type of the questionanswering system, the processing accuracy may improve even in the caseof the second example in which evaluation points are simply added up.However, even in such a question answering system, by adequatelyadjusting the value of b or k of the expression used in the thirdexample and fourth example, it is possible to obtain higher processingaccuracy than the second example in which evaluation points are simplyadded up.

In the first and fifth question answering systems, it is evident thatprocessing accuracy in the second example (conventional technique) inwhich evaluation points of answer candidates are simply added up may belower than the first example in which evaluation points of answercandidates are not counted. That is, the technique of simple additionprocessing of evaluation points has a tendency that answer candidateswith a higher frequency of appearance are likely to be selected.However, in the question answering system having good processingperformance, it is often the case that original evaluation points whichare results of extraction processing are more reliable and it is evidentthat the performance of the question answering system is contrarilylowered due to simple additions of evaluation points as in the case ofthe second example.

Especially, in the first question answering system, the third exampleshows an accuracy improvement of a maximum of 0.06 in Task1 and 0.07 inTask2 compared to the first example, that is, the example by theconventional technique and it is evident that this example has notableeffects among other examples of the present invention.

Furthermore, when the value of b according to the first counting methodor the value of k according to the second counting method ranges from0.1 to 0.5, the processing accuracy when the third example or fourthexample was used was always higher than the processing accuracy when thefirst example or second example of the conventional technique was usedexcept the second question answering system. Therefore, the value of bor k in the expression preferably ranges from approximately 0.1 toapproximately 0.5 and it is evident that the present invention canstably improve processing accuracy compared to the conventionalprocessing technique and is effective for many types of questionanswering systems.

Note that according to the conventional processing technique wherebyevaluation points of answer candidates are simply added up, when thenumber n of evaluation points is infinite, the value of added evaluationpoints also becomes infinite and the influence of addition is toostrong. In contrast, according to the present invention, when weightingis performed for each evaluation point of answer candidate, even whenthe number n of evaluation points is infinite, the value of addedevaluation points falls within the range of finite values, and thereforeappropriate weighting can be realized as the weighting according to thefrequency.

Furthermore, the present invention can be implemented as a processingprogram to be read and executed by a computer. The processing programfor implementing the present invention can be stored in an appropriatecomputer-readable recording medium such as portable medium memory,semiconductor memory and hard disk, provided by being recorded in theserecording media or provided by transmission/reception through acommunication interface using various communication networks.

The invention may be embodied in other specific forms without departingfrom the spirit of essential characteristics thereof. The presentembodiments are therefore to be considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by foregoing descriptionand all changes which come within the meaning and range of equivalencyof the claims are therefore intended to be embraced therein.

1. A question answering system which receives question statement dataexpressed in a natural language and outputs an answer to the questionstatement data from a document data group which is an answer retrievaltarget, the system comprising: answer type information storage means forstoring an answer type information that defines a plurality of answertypes representing types of language expression, one of which can be theanswer to the question statement data, and a relevancy of answer types;answer type estimating means for analyzing language expressions of thequestion statement data entered and estimating an answer type which is atype of a language expression that can be the answer to the questionstatement data based on the answer type information; document retrievingmeans for extracting keywords from the question statement data,retrieving and extracting document data including the keywords from thedocument data group; answer candidate evaluation point calculating meansfor extracting language expressions that can be an answer from thedocument data as answer candidates, determining an answer type ofindividual answer candidates and assigning evaluation points to theindividual answer candidates, the evaluation points being determined inaccordance with the answer type of the answer candidate and therelevancy of the answer types being defined in the answer typeinformation; answer candidate evaluation point counting means forsorting answer candidates having the same language expression, ranking,the evaluation points of the answer candidates having the same languageexpression in descending order calculating values of the evaluationpoints using such weighting that the value to be processed for eachevaluation point diminishes in descending ranking order of theevaluation points and a value processed for a second-placed evaluationpoint takes a value chosen in a range from 0.1 to 0.5 of the value, andregarding the sum total of the values as the evaluation point of theanswer candidate having the same language expression; and answeroutputting means for outputting answer candidates whose evaluation pointcounted by the answer candidate evaluation point counting means is equalto or higher than a predetermined evaluation as an answer.
 2. Thequestion answering system according to claim 1, wherein when countingevaluation points for each answer candidate having the same languageexpression, the answer candidate evaluation point counting meanscalculates values of the evaluation points of the answer candidateshaving the same language expression and sorted and ranked in descendingranking order of evaluation using a formula 1;Score=Σ1/(i+b)/(i+b−1)*Point_(i)  (1) where Σ is the sum total when itakes a value of 1 to n; n is the number of answer candidates having thesame language expression; and Point_(i) (1≦i≦n) is the ranking order ofevaluation points of the answer candidates; from which each sum total ofthe values of the answer candidates is obtained as the representativeevaluation point of answer candidate having the same languageexpression.
 3. The question answering system according to claim 2,wherein the answer candidate evaluation point counting means calculatesthe evaluation points using the formula 1 in which b takes a value from0.1 to 0.5.
 4. The question answering system according to claim 1,wherein when counting evaluation points for each answer candidate havingthe same language expression, the answer candidate evaluation pointcounting means calculates values of the evaluation points of the answercandidates having the same language expression and sorted and ranked indescending ranking order of evaluation using a formula 2;Score=Σk ^(i−1)*Point_(i)  (2) where Σ is the sum total when i takes avalue of 1to n; n is the number of answer candidates having the samelanguage expression; and Point_(i) (1≦i≦n) is the ranking order ofevaluation points of the answer candidates; from which each sum total ofthe values of the answer candidates is obtained as the representativeevaluation point of answer candidate having the same languageexpression.
 5. The question answering system according to claim 4,wherein the answer candidate evaluation point counting means calculatesthe evaluation points using the formula 2 where k takes a value from 0.1to 0.5.
 6. A question answering processing method which receivesquestion statement data expressed in a natural language and outputs ananswer to the question statement data from a document data group whichis an answer retrieval target, the method comprising: answer typeinformation storage processing by storing an answer type informationthat defines a plurality of answer types representing types of languageexpression, one of which can be the answer to the question statementdata, and a relevancy of an answer type; answer type estimationprocessing by analyzing language expressions of the question statementdata entered and estimating an answer type which is a type of a languageexpression that can be the answer to the question statement data basedon the answer type information; document retrieval processing byextracting keywords from the question statement data, retrieving andextracting document data including the keywords from the document datagroup; answer candidate evaluation point calculation processing byextracting language expressions that can be an answer from the documentdata as answer candidates, determining an answer type of the individualanswer candidates and assigning evaluation points to the individualanswer candidates, the evaluation points being determined in accordancewith the answer type of the answer candidate and the relevancy of theanswer types being defined in the answer type information; answercandidate evaluation point counting processing by sorting answercandidates having the same language expression, ranking the evaluationpoints of the answer candidates having the same language expression indescending order, calculating values of the evaluation points using suchweighting that the value to be processed for each evaluation pointdiminishes in descending ranking order of the evaluation points and avalue processed for a second-placed evaluation point takes a valuechosen in a range from 0.1 to 0.5 of the value, and regarding the sumtotal of the values as the evaluation point of the answer candidatehaving the same language expression; and answer output processing byoutputting answer candidates whose evaluation point counted by theanswer candidate evaluation point counting means is equal to or higherthan a predetermined evaluation as an answer.
 7. The question answeringprocessing method according to claim 6, wherein when evaluation pointsare counted for each answer candidate having the same languageexpression in the answer candidate evaluation point counting processing,evaluation points of answer candidates having the same languageexpression are sorted and ranked in descending ranking order ofevaluation, and a formula 1;Score=Σ1/(i+b)/(i+b−1)*Point_(i)  (1) where Σ is the sum total when itakes a value of 1 to n; n is the number of answer candidates having thesame language expression; and Point (1≦i≦n) is the ranking order ofevaluation points of the answer candidates; which is calculated toobtain each sum total of the values of the answer candidates as therepresentative evaluation point of answer candidate having the samelanguage expression.
 8. The question answering processing methodaccording to claim 6, wherein when evaluation points are counted foreach answer candidate having the same language expression in the answercandidate evaluation point counting processing, evaluation points ofanswer candidates having the same language expression are sorted andranked in descending ranking order of evaluation, and a formula 2;Score=Σk ^(i−1)*Point_(i)  (2) where Σ is the sum total when i takes avalue of 1 to n; n is the number of answer candidates having the samelanguage expression; and Point (1≦i≦n) is the ranking order ofevaluation points of the answer candidates; which is calculated wheretoobtain each sum total of the values of the answer candidates as therepresentative evaluation point of answer candidate having the samelanguage expression.